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some shape information is available from an indepen- 
dent source. The optical image provides the indepen- 
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METHOD AND APPARATUS FOR SENSOR 
FUSION 

ORIGIN OF THE INVENTION 5 

The invention described herein was made by an em- 
ployee of the United States Government, and others, 
and may be manufactured and used by and for the Gov- 
ernment of the United States of America for govern- 
mental purposes without payment of any royalties 10 
thereon or therefor. 

BACKGROUND OF THE INVENTION 

1. Field of the invention 

The present invention relates to image enhancement 15 
methods and more particularly to image enhancement 
methods using the fusion of data from optical and mi- 
crowave sensors. 

2. Description of the Related Art 

Related inventions include the following U.S. Pat. 20 
Nos. 4,672,562 and 4,672,564 to Egli, et al, which dis- 
close photodetector arrays and include computational 
means for determining spatial information about target 
objects; U.S. Pat. No. 4,620,285 to Perdue which 
teaches a combined sonar ranging and light detection; 25 
U.S. Pat. No. 3,981,010 to Michelsen which shows a 
system for locating objects and includes a radar system 
in combination with a television camera having the 
outputs combined in a data processing system to deter- 
mine direction and range to the objects; U.S. Pat. No. 30 
4,611,292 to Ninomiya, et al, which discloses a robot 
vision system including twin light beams used to deter- 
mine the coordinates of an object; U.S. Pat. No. 
4,550,432 to Anderson which teaches an image proces- 
sor using a moment generator including processor 35 
means for determining the moments of a geometrical 
object; U.S. Pat. No. 4,443,855 to Bishop, et al, which 
reveals a sensing and control system using a mask algo- 
rithm image processor to generate control or command 
signals. None of the discovered related art teaches use 40 
of a first sensor in predicting the output of a second 
sensor and use of non-linear minimization techniques in 
determining the shape of objects. 

Sensor Fusion (SF) techniques have been developed 
to combine information from such diverse sources as 45 
optical imagery, laser ranging, structured light, and 
tactile feedback. The present invention relates to the 
fusion of a set of polarized RCS measurements with 
optical imagery. The polarized RCS yields detailed 
information about an object’s surface only if some other 50 
independent surface information is available. An inter- 
preter needs information from an alternate source such 
as an optical image to convert the RCS into spatial 
information that a general SF system can use. Once this 
conversion is completed, the more traditional concepts 55 
of SF may be employed. A brief examination of the 
state-of-the-art in SF is provided, along with compari- 
sons between the state-of-the-art and our SF method. 

Sensor Fusion Generally ^ 

The purpose of SF is to combine the interpreted 
outputs of various sensors into a consistent world-view 
that is in some way better than its component interpre- 
tations, for example, the sense of confidence in an inter- 
pretation may be greater, or the resulting composite 65 
surface or workspace may be more complete than for a 
single sensor, or a combination of both. Although all SF 
systems produce some kind of spatial, or geographic 
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information, they may be divided into two groups. The 
first type of fusion system sees the world as a collection 
of discrete objects and tries to localize these objects. 
The second type attempts to describe the details of 
continuously connected surfaces. The common thread 
is the attempt to deal with uncertain, conflicting, and 
incomplete data. Most SF systems attempt to sift 
through a collection of tokens representing spatial prim- 
itives and, when possible, merge two or more tokens 
into one. Examples of the tokens used are frames de- 
scribing objects and individual surface patch contour 
descriptions. 

Various techniques have been developed to perform 
the fusing of information from different sensors that 
describe the same object. Harmon, et al, (S. Y. Harmon, 
G. L. Bianchini, and B. E. Pinz, “Sensor Data Fusion 
Through a Distributed Blackboard,” Int. conf. on Ro- 
botics and Automation, pp. 1449-1454, 1986) divides the 
approaches into three categories: “averaging”, “decid- 
ing ”, and “guiding”. In “averaging” techniques, confi- 
dence measures are used to weight various estimates of 
the same property to compute an average value that 
may not be exactly equivalent to any individual esti- 
mate. When “deciding”, one particular estimate is 
picked from many others to represent the entire data 
set. Again, this choice is based on confidence measures. 
The third technique, “guiding”, uses one estimate to 
direct the acquisition of further estimates. 

The present invention is SF method employing pri- 
marily a “guiding” shape extraction technique. The 
partial surface model acquired from optical image data 
guides the conversion of RCS data into a complete 
surface model. 

Object Localization Techniques 

Researchers have investigated a variety of statistical 
methods for- integrating uncertain and redundant geo- 
metric information about discrete objects. These meth- 
ods include weighted estimates, confidence measures 
and Bayesian estimation. Other researchers have con- 
centrated on the overall architecture of the resulting 
information-processing system. Both aspects of the 
problem must be addressed in order for an efficient 
knowledge and information handling mechanism to use 
the most accurate statistical model. 

Statistical Models 

Shekhar, et al, (S. Shekhar, O. Khatib, and M. 
Shimojo, “Sensor Fusion and Object Localization”, Int. 
conf. on Robotics and Automation, pp. 1623-1628, 
1986) have developed a system for determining the six 
degrees of freedom of an object by combining multiple 
estimates. The sensory data considered was from tactile 
sensors placed on the robot end-effector. Errors in the 
positioning of the manipulator lead to uncertain esti- 
mates of attitude and position. These measurement er- 
rors are assumed to be known a priori and are used to 
weight final estimates. Orientation and position esti- 
mates are carried out independently. Position estimates 
are computed by a simple weighted average. Orienta- 
tion estimates are derived by representing the rotation 
matrix as a quaternion and solving the resulting linear 
system of equations by a weighted left inverse. This 
method assumes much information is provided to the 
system. This information includes complete stored ob- 
ject descriptions and correspondence between sensed 
points and model points. This technique is clearly of the 
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“averaging” variety. A complete sensor fusion system is 
described by Luo, et al. (Ren C. Luo, Min-Hsiung Lin, 
and Ralph S. Scherp, “The Issues and Approaches of a 
Robot Multi-Sensor Integration,” Proc. IEEE Int. conf. 
on Robotics and Automation, pp. 1941-1946, Raleigh, 5 
N. C., 1987). The decision process and the statistical 
models are considered together. A group of sensor 
observations is first selected based on the task at hand. 
Observations of the same physical property are fused by 
a two-step process. The first step selects observations to 10 
be fused, and the second step combines them into one 
estimate. Each observation is characterized by a normal 
power distribution function (p.d.f.), and a distance is 
defined between p.d.f.’s. These distances ar then thre- 
sholded and the largest connected group of observa- 15 
tions is chosen to be fused. The optimal estimate is then 
derived by maximizing the sum of conditional probabili- 
ties for the estimate weighted by the probability of each 
observation. Finally, an attempt is made to compensate 
those observations discarded in the first step of the 20 
process. This technique constitutes a hybrid between 
“deciding” and “averaging”. 

A complex method for integrating disparate sensor 
observations is presented by Durrant- Whyte (Hugh F. 
Durrant-Whyte, “Consistent Integration and Propaga- 
tion of Disparate Sensor Observations,” Proc. IEEE 
Int. conf. on Robotics and Automation, pp. 1464-1469, 
1986). Uncertain measurements are combined in such a 
way that geometric consistency is maintained in the 
world-model. To insure robustness, the p.d.f.’s of each 
sensor observation are characterized as the sum of two 
normal distributions, of which, only one covariance is 
known. Observations that fall outside an ellipse enclos- 
ing the mean of the known distribution are discarded. 
The remaining samples are considered jointly normal 
with known covariance. A Bayesian estimate of the 
sensor value is found by minimizing a loss function 
under geometric constraints. Durrant-Whyte empha- 
sizes that he is not solving a recognition problem. The 
SF system deals only with abstract geometric informa- 
tion. This system is also a combination of “averaging” 
and “deciding”. 

Computing Architectures 

Various architectures have been proposed for han- 
dling multiple sensory information. Harmon, et. al. de- 
scribe a system based on a distributed blackboard. Each 
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dictions about the world and then matching these pre- 
dictions with the observed data. The system deals with 
dynamic information in real time, and is updated contin- 
uously. 

Chiu, et. al. claim that sensor fusion should be repre- 
sented hierarchically as a data-flow process (S. L. Chiu, 
D. J. Morley, and J. F. Martin, “Sensor Data Fusion on 
a Parallel-Processor,” Int. conf. on Robotics and Auto- 
mation, pp. 1629-1633, 1986). They propose that this 
process be implemented on a parallel processor. 

Surface Reconstruction Techniques 

The RCS-Optical system of present invention falls 
into the category of SF techniques intended to recon- 
struct continuous surfaces rather than collections of 
discrete objects. Previous work in this area has been 
done by Allen and Bajcsy, Grimson, Crowley, and 
Wang and Aggarwal. 

Allen and Bajcsy (P. Allen and R. Bajcsy, “Two 
Sensor Are Better than One: Example of Integration of 
Vision and Touch,” in Proc. Third Int. Symposium on 
Robotics Research, Giralt, Eds., MIT Press: Cam- 
bridge, Mass., 1985) demonstrated that the combination 
of multiple sensors can produce a 3D object description 
that is better than those derived from individual sensors. 
They have used the paradigm of computational stereo 
to build occluding contour descriptions that contain 
gaps and inaccuracies. The interior of the surface and 
the uncertain points on the contours are filled in with an 
active tactile sensor. Coons patches (composite bicubic 
surfaces) are used to interpolate between critical points. 
The resulting object description is a 2J D sketch of the 
unknown object. This is a good example of a “guiding” 
technique, since the optical information controls the 
movement of the tactile sensor. 

By restricting unknown objects to those composed of 
polyhedral faces, Grimson, et al (W. E. L. Grimson, and 
T. Lozano-Perez, “Model Based Recognition and Lo- 
calization from Sparse Range and Tactile Data,” Int. 
Jml. of Robotics Research, Vol. 3, no.3, Fall, 1984) 
were able to generate some constraints that allowed 
sparse isolated surface points and normal estimates to be 
matched with stored models. The surface data could, in 
principle, be derived from any source, the specific cases 
being range and tactile observations. Using heuristics 
derived from the geometry of polyhedral objects, such 
as maximum and minimum distances between pairs of 


sensor subsystem has its own copy of the up-to-date points on different facets, consistent normal angles, 

world model, made up of tokens, each of which repre- amounts of rotation, etc., the investigators were able to 


sent s an object, a selected object property and its value, 50 prune the search tree of point-facet pairings. With the 
along with an error range, a confidence factor and a pruning done beforehand, most point labelings need not 

time stamp. Communication between subsystems is be considered when determining a proper model match, 

accomplished through a local area network. Subsystems Grimson found that even a poor estimate of surface 

share only high-level, abstract information, leaving the normals greatly simplifies the attitude determination 

recognition task to the individual sensors. 55 problem. In a later publication, they expand their 

A system using two types of knowledge representa- method to include recognition of partially occluded 

tion is described by Kent, et. al (E. W. Kent, M. O. objects (W. Eric L. Grimson and Tomas Lozano-Perez, 

Shuier, and T. H. Huang, “Building Representations “Search and Sensing Strategies for Recognition and 

from Fusions of Multiple Views,” Proc. IEEE Int. conf. Localization of Two- and Three-Dimensional Objects,” 

on Robotics and Automation, pp. 1634-1639, 1986). 60 Third Int. Symp. on Robotics Research, pp. 73, 80, MIT 
The purpose of this system is to fuse multiple views of Press, Cambridge, Mass., 1986). Also, the efficiency of 

the same workspace taken over time. A world model is the algorithm was improved by estimating possible 

built up and compared to internally stored information. configurations by Hough transforms. 

The system is also capable of handling objects that have Crowley (J. L. Crowley, “Representation and Main- 
no stored model. Spatial information about the work- 65 tenance of a Composite Surface Model,” Int. conf. on 
space is represented by an octree. Knowledge about Robotics and Automation, pp. 1455-1462, 1986) saw the 

discrete objects and their features is maintained sepa- need, when combining information from different 

rately in a list. The system operates by generating pre- sources, for a standard knowledge representation. He 
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used primitives representing planar surface elements 
and contours joining elements. Each attribute of a prim- 
itive is represented as an uncertainty range in which the 
exact number is believed to lie. The result is surface 
elements and contours that are not two* and one- 5 
dimensional manifolds in three-space, but represent 
some volume where such manifolds may exist. Both 
stored knowledge and acquired data may be repre- 
sented in the same form. A model that is being con- 
structed need not be complete, or represent any physi- 
cally realistic surface. A surface patch need not have 
bounding contours on each side, and a contour does not 
need to point to two surface patches. These missing 
facts may be inferred, however, in the fusion process, 15 
and incomplete models matched to stored library mod- 
els. The form of each piece of data remains intact as the 
surface model is constructed and primitives are added 
and merged. Each primitive has a confidence factor 
associated with it, and elements are combined or re- 20 
moved based on this confidence factor. This SF tech- 
nique is of the “deciding” variety. 

Wang and Aggarwal also dealt with modeling 3D 
surfaces using diverse sources of information (Y. F. 
Wang and J. K. Aggarwal, “On Modelling 3D Objects 25 
Using Multiple Sensory Data,” Proc. IEEE Int. conf. 
on Robotics and Automation,: pp. 1098-1103, Raleigh, 

N. C., 1987). Their technique is similar to ours in that 
they used one source of information to determine oc- ^ 
eluding contours, and another source to fill interiors of J 
3D objects. The occluding contours are derived from a 
thresholded optical image, and partial surface structures 
are inferred from structured light. Multiple views of an 
object are considered. The partial surface structures are 35 
allowed to move along an axis defined by the occluding 
contours observed at the same time. The actual position 
of any one surface along its axis is determined by match- 
ing it with the cylindrical volume derived from occlud- 
ing contours in another view. Pairs of surfaces and 40 
bounding volumes that are most nearly orthogonal to 
each other are combined. This allows one surface struc- 
ture to be positioned to the greatest accuracy without 
relating to other surface structures. 

Wang and Aggarwal mention that efficient data 45 
structures such as octrees are used to store the surface 
data, but the issue of uniform data representation was 
not directly addressed in their paper. It is possible that 
an octree structure may have to be altered as new sur- 
face data is incorporated at different levels of resolu- 50 
tion. This may result in some wasted time in a general 
SF system. Most SF systems use lists of tokens of equal 
significance that may be linked bidirectionally in vari- 
ous ways as opposed to an hierarchical data structure ^ 
such as an octree. As new information is added in this 
type of system, relationships may be noted without 
disturbing the links that have already been established. 
The final smooth surface representation is derived by 
spline approximation. Therefore, the spline basis func- 
tions and their knots may be thought of as the surface 
knowledge representation primitives. If conflicting sur- 
face points arise, their weighted average is computed. 
Thus, this SF technique falls into the “averaging” cate- 
gory. It might also be classified as a “guiding” tech- 65 
nique, since the occluding contours from one view are 
used to guide the placement of partial surface structures 
in another. 
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RCS/Image Surface Reconstruction 

Examining these previous attempts at sensor fusion, 
we are able to make comparisons and draw suggestions 
from the work that has gone before. Even though our 
system uses a new combination of information, there 
remain strong similarities with other techniques. In 
addition there are shared pitfalls. Although our micro- 
wave and video SF system may at first appear to be a 
simple “guiding” technique, the problem of how to 
handle conflicting and uncertain information had to be 
addressed. If a partial surface may be reconstructed 
from the RCS, how can it be resolved with the optical 
image surface? Are there points of contradiction? 
Which interpretation should be given more weight, and 
how are such weights assigned? Also, what is the most 
efficient and uniform data structure for storing, infer- 
ring, and reconstructing surface-knowledge derived 
from the optical image, RCS, and library models? 

In one particular application of our SF technique, 
interior surface patches are reconstructed from the RCS 
by an iterative minimum error technique. This surface 
patch covers a portion of the surface whose optical 
image has been degraded in some fashion. The surface 
patches to be reconstructed have some predetermined 
finite support, such as a rectangle, in the parameter 
domain. It is unlikely that the degraded portion of the 
image will have this exact shape, so the smallest support 
of predetermined shape that completely covers the 
degraded portion must be used. This results in overlap- 
ping areas between the optically derived surface and the 
RCS reconstructed surface. Although the two surfaces 
may agree at the initial estimate, the minimization pro- 
cedure could easily produce a surface patch that does 
not match the optical surface exactly at its edges. Since 
both surfaces are uncertain to some degree, some con- 
vention must be adopted to resolve these conflicts. If 
the surface is represented by a continuous manifold, an 
“averaging” technique must be used to preserve conti- 
nuity. If, however, the surfaces are represented by a 
collection of discrete points, a decision process may be 
applied to choose an appropriate z value at a given 
location. The “averaging” step that preserves continu- 
ity could then be taken care of by spline interpolation. 
The weight or confidence assigned to each point de- 
pends on a variety of possible sources of error. For the 
optically derived surface, uncertainties may arise from 
errors in camera positioning, light source location, 
noise, model inaccuracy, and image registration. For 
the RCS surface, factors such as detection error, ambi- 
ent reflections, and range uncertainty can contribute to 
RCS measurement error, and inappropriate modeling 
assumptions, numerical inaccuracies, and insufficient 
grid density can produce error in the conversion from 
RCS to surface. 

In simulations of this method, discrete surface point 
samples were used as a means for representing 3D geo- 
metric knowledge. This type of representation lends 
itself well to microwave scattering computations. In 
addition to a location in three-space, each surface point 
should carry with it an estimate of surface normal at 
that point. If no surface normal estimate is available, a 
coarse one may be inferred from the surrounding points. 
A token that represents a surface point must also carry 
a confidence factor appropriate to the decision process 
to be used. Also, a range of possible values may be 
specified for each numerical attribute in the manner 
adopted by Crowley. This additional information is 



5 , 005,147 


7 

necessary in order to resolve inconsistencies between 
the optical surface and the final surface derived through 
the error minimization process. 

Utility of Sensor Fusion 5 

One example of the utility of incorporating the micro- 
wave RCS into a space robot’s sensory information is 
attempting to overcome some of the difficulties associ- 
ated with optical imaging in space. 

A sensor system for a space robot must be able to 10 
gather and interpret information about the spatial loca- 
tion of discrete objects in a scene, their shape and mo- 
tion. This information must be complete and accurate 
enough to allow the robot to navigate within its work- 
space, and manipulate selected objects. Clearly, the 15 
demands on a space robot sensor system are dictated by 
the type of scene that it expects to encounter. Further- 
more, the scene type is dictated by the task that the 
robot must perform. 

The shape extraction system makes use of the assump- 20 
tion that targets consist of isolated, smooth shapes con- 
structed of some perfectly conducting material. In real- 
ity, more than one target may be present, and the shape 
of these targets may be complex. Certain questions must ^ 
be answered about the scene to be viewed, and the 1 
objects that make up these scenes. For example, their 
expected complexity, range, and motion must be deter- 
mined. 

The attributes of the individual objects that comprise 3Q 
the scene are also of interest. Target properties that 
affect the accuracy of the scattering model are shape, 
surface roughness, dielectric constant, and motion. Ob- 
jects may in turn be decomposed into components that 
have well-defined properties. 35 

Typical scenes which an automated sensor system 
might encounter in space vary widely. They may be 
simple, or complex in terms of numbers of objects, and 
they may be in motion with respect to several degrees 
of freedom. In general, the scene characteristics depend 40 
on the task at hand. The characteristics that must be 
determined for each task are complexity, range, and 
motion. 

Jobs that could be automated in space usually fall into 
the three broad categories of tracking, retrieval, and 45 
servicing. Each of these tasks involve different types of 
scenes which a robot’s sensing system must deal with. 
Over the life of a given robot mission all three tasks 
could be encountered, but a single vision system should 
only have to interpret one type of scene at a time. Con- 50 
sider an automated satellite maintenance mission. Such 
a task requires that the robot first track the target from 
a distance, then approach the target at close range, and 
finally dock and execute the necessary servicing opera- 
tions. 55 

An important aspect of the shape reconstruction 
method of instant invention is the use of optical sensing 
to derive a first approximation to the target shape. A 
computer vision system that is able to even partially 
determine the shape of three-dimensional objects will 60 
consist of many independent sub-systems. Each of these 
subsystems may be the result of years of research and 
testing, so the literature on this subject is vast. 

Computer vision system components usually fall into 
two categories: those that deal with low-level pro- 65 
cesses, and those that perform high-level functions. 
Low-level vision includes image acquisition, digitaliza- 
tion, and simple linear filtering. High-level vision in- 
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volves the more complex tasks of matching and shape 
determination. 

Low-Level Vision Subsystems 

Image acquisition is the first step in the optical imag- 
ing sequence. In this step, the light that comprises the 
image enters the camera through a series of lenses, and 
is focused onto a sensing element. The sensing element 
converts the light into an electrical signal. Important 
characteristics of this stage in the image formation pro- 
cess are the aspect ratio of the camera, the focal length 
and diameter of the optical lenses, and the type and 
characteristics of the light sensor. 

The analog video image must then be converted into 
digital form so that the optical information may be 
manipulated by computer. This process, known as digi- 
talization, also affects the usefulness of the final image 
product. Parameters such as the size, number, and spa- 
tial distribution of the pixels influence the amount of 
information from the analog image which survives into 
the digital image. 

Low-level vision also includes some simple filtering 
techniques. These traditional image processing routines 
are important to the resulting shape interpretation and 
must be given careful consideration. The two low-level 
processing steps that are relevant to the present inven- 
tion are thresholding and edge detection. A vision sys- 
tem must be able to separate objects from background 
to approximate their shape for the initial parameter 
determination. This is most easily accomplished 
through thresholding, where a bright object is assigned 
one brightness value, and the dark background another. 
In some instances, the numerical shape reconstruction 
system depends heavily on knowledge of the scattering 
target’s occluding contours. To determine an object’s 
occluding contours, edge detection is required. 

High-Level Vision Subsystems 

High-level vision subsystems are those that not only 
process the optical image, but attempt to extract some 
information regarding the location, shape, motion, etc. 
of the objects in the image. There are two approaches to 
this problem. The first approach is to compare the opti- 
cal image to a library of stored images, or images gener- 
ated from a library of stored object models. The second 
approach is to generate a three-dimensional shape di- 
rectly from the information in the optical image. The 
matching approach is much more reliable, but requires 
a large library of stored models, and a vast amount of 
knowledge about the various images these models may 
generate. 

The second approach is manifested in such tech- 
niques as shape-from-shading and photometric stereo. 
These “shape-from” techniques are limited by the qual- 
ity of illumination available. They have only proven 
successful in controlled experiments with simple shapes. 
Also, the “shape-from” techniques only generate sur- 
face normals. Reconstructing a unique surface shape 
from these normals is not always possible. 

Another method for generating shape directly from 
the optical image is stereo vision. In stereo vision, simul- 
taneous images from cameras separated by a small dis- 
tance are correlated so that points in the two images 
that correspond to the same physical location are 
matched. From knowledge of camera placement and 
the disparity between two matched points, the 3 D loca- 
tion of the corresponding physical point can be com- 
puted. The most difficult aspect of this procedure is the 
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correlation between images. Seldom is a perfect match 
found throughout the image, and specular points pres- 
ent particularly troublesome matches. Stereo is, how- 
ever, a very promising 3D vision technique and most of 
the space robotic vision systems under design make use 5 
of it. 

Application of Sensor Fusion 

A free-flying space robot needs detailed information 
about the location of objects in its workspace and the 10 
shape of individual objects. The most common sensors 
for acquiring this information are electronic cameras 
sensitive to optical wavelengths. The space environ- 
ment, however, presents unique problems to an optical 
3D shape sensing system. The lack of atmosphere ere- 15 
ates deep shadows and intense illumination. Specular 
points become very bright, and shadowed edges are 
indistinguishable against dark backgrounds. These 
problems cannot always be easily overcome by simple 
image processing or enhanced vision methods. Sensory 20 
data from some independent physical regime must be 
used to augment a purely optical robot sensor system. 

The goal of a sensor fusion system may be to localize 
objects within a robot workspace, or to determine the 
shape of individual objects. The application considered 25 
here is extracting the shape of 3D objects. Various 
sensors which have been studied by researchers for 
robotic applications include multiple camera views, 
multiple processes on a single view, tactile sensors, laser 
range images, and ultrasonic imaging. Our invention has 30 
focused on the fusion of camera images with electro- 
magnetic scattering data of conventional bandwidths 
(not high resolution) and frequencies, i.e. X band, Ku 
band, etc., to determine the shape of remote 3D objects 
in space. 35 

Ideally, a robot working in space would use high- 
resolution microwave imaging to augment its tactile 
and optical sensors. This type of system requires large 
bandwidth, multiple viewing angles, and sophisticated 
signal processing, and produces an image consisting of 40 
intense regions at points of specularity. The technology 
and equipment required for such a system is, however, 
prohibitively cumbersome and expensive for many ap- 
plications, and interpretation techniques lag behind the 
imaging technology. An alternative to microwave im- 45 
aging is the use of polarized radar scattering cross-sec- 
tions (RCSs). The RCSs do not yield high-resolution 
images, but the analysis of depolarization can provide 
important shape information about the scattering sur- 
face; especially at the specular point, where the optical 50 
image often fails. To circumvent the ambiguity of the 
RCS, the incomplete optical image can be used to guide 
the conversion of RCS into shape. The resulting scatter- 
ing surface shape description is more complete than can 
be derived from the RCS or the camera image alone. 55 

The main contribution of this investigation has been 
the development of numerical methods that determine 
some characteristics of a surface from an incomplete 
optical image and the observed RCS. This is done by 
modeling the unknown characteristic of a surface by 60 
some parameter vector and applying a non-linear-least- 
squares algorithm to minimize the difference between 
the observed RCS and a theoretical RCS that results 
from the parameterized surface model. The optical 
image is used to provide an initial estimate of the param- 65 
eterized surface. The success of such a method depends 
to a large extent on the accuracy of the theoretical RCS 
model used. Models using the geometrical theory of 
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diffraction, moment methods, and physical optics have 
been considered. Another limitation on the success of 
the method is the type and quality of information input 
to the system. However, in a typical application a con- 
figuration such as the one shown symbolically in FIG. 2 
may be used. 

An application of particular interest is one in which 
the method of moments is used to model the RCS. Al- 
though this involves the numerical solution of a com- 
plex integral equation, it is significant because it allows 
the computation of microwave scattering from arbi- 
trary surfaces. Simpler modeling techniques such as the 
geometrical theory of diffraction or physical optics 
constrain the scattering target to simple geometric 
shapes such as cylinders and plates. The method of 
moments requires that the currents on the surface of the 
scattering object be expanded in some orthonormal 
basis. Without some independent shape information, no 
constraints exist on the shape and configuration of these 
basis functions. This information must be derived from 
the optical image. It can be shown that non-linear error 
minimization employing the method of moments is sim- 
plified if the shape and configuration of the basis func- 
tions remain fixed throughout the procedure (See, “Mi- 
crowave and Optical Sensor Fusion for the Shape Ex- 
traction of 3D Space Objects”; a doctoral Thesis of 
Scott W. Shaw, Rice University; Houston, Tex., April, 
1988). These observations indicate that the optical 
image is essential to the feasibility of this technique. We 
have employed the assumption that the occluding con- 
tours are known at the outset and can be extracted from 
the camera data using standard image processing tech- 
niques. 

Several simulations and experiments were performed 
demonstrating the application of the RCS/image sensor 
fusion system. The invention involves a method for 
building a three-dimensional (3D) surface model of a 
remote object from a single digital camera image and a 
set of polarized radar scattering cross-sections. As pre- 
viously stated, in space, camera images are difficult to 
analyze because light does not behave as it does in the 
atmosphere. Intensely bright areas and deep shadows in 
the image create problems for standard shape extraction 
techniques such as shape-from-shading, photometric 
stereo, and stereopsis. Often the resulting 3D surface 
model is incomplete. Our technique for improving this 
surface model is to use it as a first approximation in an 
iterative scheme that minimizes the difference between 
theoretically predicted cross-sections and observed 
cross-sections. We have demonstrated this technique on 
three problems. The first was to determine a missing 
edge of a flat plate. The second was to determine the 
size and rotation in the horizontal plane of an ellipsoid. 
The third was to recover the shape of an interior surface 
patch on an arbitrarily shaped scattering target. 

Application 1 

This application involved determining an edge of a 
perfectly conducting, rectangular, flat plate that has 
been lost in shadow. The steps in the process are: 

1. Obtain an approximate orientation of the plate by 
stereopsis. 

2. Enhance edges in the camera image. 

3. Threshold the edge-enhanced camera image. 

4. Extract the three visible boundaries of the plate 
from the thresholded camera image. 

5. Determine the width of the plate and, hence, the 
missing edge, using a GTD scattering model to generate 
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theoretical radar cross-sections of the plate (see R. A. 
Ross, “Radar Cross Sections of a Rectangular Flat Plate 
as a Function of Aspect Angle”, IEEE Transactions on 
Antennas and Propagation, Vol.AP-14, No. 3, 
pp329-335, May, 1966), and minimizing the difference 5 
between these theoretical cross-sections and the ob- 
served cross-sections using the program “lmdifO” 
found in the MINPACK group of programs published 
by Argonne National Laboratories. 

* i. . * 10 

Application 2 

The second application was to build a model of a 
perfectly conducting ellipsoid rotated in the horizontal 
plane. The following steps describe this application: 

1. Enhance the edges of the camera image. 15 

2. Threshold the edge-enhanced image. 

3. Determine the vertical axis of the ellipsoid directly 
from the thresholded, edge-enhanced image. 

4. Determine the apparent width of the ellipsoid in 
the horizontal plane from the thresholded, edge- 20 
enhanced image. 

5. Determine the other two axes and the rotation in 
the horizontal plane by generating a geometrical optics 
cross-section (see G. T. Ruck, D. E. Barrick, W. D. 
Stuart, and C. K. Kirchbaum, Radar Cross-section 25 
Handbook, Vol. 1, Plenum Press, New York, 1970), a 
first order approximation to the cross-polarized cross- 
section (see S. J. Choudhuri and W. M. Boemer, “A 
Monostatic Inverse Scattering Model Based on Polar- 
ized Utilisation,” Appl Phys., Vol.l 1, pp337-350, 1976), 30 
and minimizing the difference between the theoretically 
generated cross-sections and the observed cross-sec- 
tions, again using the “lmdifO” routine. 

Application 3 35 

This application involved building a model of an 
arbitrary surface given the scattering cross sections and 
a degraded optical image. The image is degraded only 
around the specular point of the scattering object. The 
reconstruction procedure is: 40 

1. Using shape-from-shading, photometric stereo, or 
streopsis, construct a model of the visible portion of the 
target surface everywhere except in a patch surround- 
ing the specular point of the target. 

2. Describe the specular patch of the model by a 45 
spline surface. 

3. Determine the exterior knots of the spline patch 
from the camera-generated model. 

4. Solve for the interior knots of the spline patch by 
generating co-polarized and cross-polarized scattering 50 
cross-sections by a moment-methods technique (see R. 

F. Harrington, Field Computation by Moment Methods. 

The Macmillan Company, New York, 1968), and mini- 
mizing the difference between these theoretical cross- 
sections and the observed cross-sections, again using the 55 
“lmdifO” routine. A reliable moment-method code is 
the “Numerical Electromagnetic Code,” published by 
Ohio State University. 

The first two applications included experimental data 
as well as simulations, whereas the third was simulated 60 
only. The results show that surfaces can be recovered 
with very little error when the polarized RCS and cer- 
tain parameters from the camera image can be measured 
accurately. 

In developing the invention, it was observed that 65 
optical sensing in the space environment does not al- 
ways provide enough information to reconstruct 3D 
surfaces. The fusion of radar scattering data with the 


camera images was proposed as a solution to this prob- 
lem. Specifically, the use of polarized radar scattering 
cross-sections was suggested. It is an objective of the 
invention to produce sensor fusion resulting in work- 
space interpretation which is better than the component 
interpretations in the sense of either confidence or com- 
pleteness. The goal was to show that such a fusion is 
possible. Certainly, fusion was achieved. The complete 
surfaces were indeed recovered from the component 
interpretations. Also, it could be said that the confi- 
dence in the final surface was increased over the indi- 
vidual components. The component interpretations 
cooperated to achieve a correct solution, i.e. the inde- 
pendent sensors perceived the same physical object. 

The overall favorable results of the applications, 
however, should be interpreted in light of the simplifi- 
cations that were made. First of all, the problems were 
designed with the application in mind, also, these were 
not the most difficult situations imaginable. In particu- 
lar, the plate and ellipsoid were scattering geometries 
which would probably seldom occur in a real space 
robotic situation, on the other hand, many complex 
scattering geometries can be approximated as collec- 
tions of smaller, simple objects. The cylinder, in particu- 
lar, has a well-known and easily predictable RCS; it is 
one of the most thoroughly studied scattering geome- 
tries in existence since exact solutions to the wave equa- 
tion are available for this case. The RCS of spacecraft, 
satellites, etc. are often computed by a scattering code 
which models complex objects as connected cylinders 
of various shapes. Thus, recovering the shape of simple 
scattering geometries is a first step to developing solu- 
tions for more complex geometries. The plate was 
chosen for its well-known and accurate closed-form 
RCS expression, but the ellipsoid has no such expres- 
sion. The ellipsoid remains one of the most difficult 
scattering problems, and the prediction of diffraction 
patterns for general ellipsoids is still a research problem. 
It is encouraging, therefore, that good results were 
obtained for the ellipsoid, even though gross simplifica- 
tions were made in computing the RCS. The time- 
domain scattering results were essential here, since the 
specular return, for which the simplifications hold, 
could be separated from the creeping wave reflection. 

The hidden simplification in the design of the arbi- 
trary surface experiment is that the surface used to gen- 
erate the observations was constructed in the same way 
as the approximation surface. Splines of the same degree 
were used to define both surfaces, so an exact solution 
was always possible. A better test of this method would 
be to construct the surface used to generate the observa- 
tions from splines of a higher degree than those used to 
model the scattering response. Alternately, the observa- 
tion-generating surface could be defined arbitrarily, 
with no reference to splines, perhaps containing sharp 
corners or other features that cannot be exactly repre- 
sented by continuous functions. The results of a spline 
approximation in such a case would reveal insights 
about the usefulness of the shape reconstruction tech- 
nique applied to truly arbitrary surfaces. 

Another simplification that affected the outcome of 
the experiments is the assumption that the scattering 
targets are perfectly conducting. In practice, this condi- 
tion is seldom met, although it may be approximated in 
certain space objects. If the target is not perfectly con- 
ducting, its dielectric constant must be determined 
along with the target shape. In addition, the scattered 
field expressions increase in complexity. While this 
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difficulty is not insurmountable, its addition increases 
the number of surface parameters that must be deter- 
mined, and therefore places more demands on the num- 
ber of observations. 

One last assumption that was made and that is seldom 5 
achieved in a real space situation is target stationary. 
Usually, objects are rotating about some axis and mov- 
ing translationally with respect to the sensor. This cre- 
ates Doppler shifts in received radar frequency, and 
target scintillation. While Doppler shift is desirable in 
target tracking situations, it creates problems for RCS 
measuring systems. One way in which this might be 
overcome is to use a laser range sensor to determine 
target velocity and predict Doppler shift. 

In the flat plate and arbitrary surface simulations it 15 
was seen that the purely numerical error minimization 
procedure alone did not always converge to the correct 
solution. This was overcome by comparing the residual 
error at convergence to a threshold related to the noise 
variance of the data. If the residual was greater than the 20 
threshold, the initial parameter was perturbed slightly 
and the numerical procedure started again. This need 
for supervision in order to ensure global convergence 
even in the simplest case points out the limitations of a 25 
purely numerical algorithm. In order to implement such 
adjustment a supervisory system would have to be in- 
corporated in a given application as represented sym- 
bolically by the intelligent module of FIG. 2. 

Overall, the results of the our work demonstrate that 
there is merit in using information from some indepen- 
dent source to guide the inversion of electromagnetic 
data. Previous work in shape determination from scat- 
tering data has concentrated on the scattered electro- 
magnetic field as the sole source of information. Much 35 
of this previous work consisted of attempts to regularize 
the ill-posed inversion problem The ill-posed nature of 
the inversion problem stems from the lack of complete 
information about the scattered field. If information 
from some other physical regime is available about the 40 
scattering target it can be used to complement the lim- 
ited scattered field description. In our case, this has been 
accomplished by using the camera information to re- 
duce the number of unknown surface parameters. 

SUMMARY OF THE INVENTION 45 

The invention comprises method and apparatus for 
fusion of data from optical and radar sensors by error 
minimization procedure. The method has been applied 
to the problem of shape reconstruction of an unknown 50 
surface at a distance. The method involves deriving a 
surface model (which may be incomplete) from an opti- 
cal sensor. The unknown characteristics of the surface 
are represented by some parameter. The correct value 
of the parameter is computed by iteratively generating 55 
theoretical predictions of the Radar cross-sections 
(RCS) of the surface, comparing the predicted and the 
observed values for the RCS, improving the surface 
model from results of the comparison, and repeating the 
process until the difference between the predicted and 60 
observed values are below an acceptable threshold. 
Theoretical RCS may be computed from the surface 
model in several ways. One RCS prediction technique is 
the method of moments. The method of moments can 
be applied to an unknown surface only if some shape 65 
information is available from an independent source. 
The optical image provides the independent informa- 
tion. 
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The present invention presents a new system for ex- 
tracting the 3D shape of space objects using optical 
images and the polarized RSC. The system works by 
describing the unknown surface parametrically and, 
using electromagnetic modeling techniques, minimizing 
the error between the predicted and the observed polar- 
ized RCS set. The optical image is incorporated by 
providing an initial estimate of the scattering surface 
shape. Obviously, the success of such a method is highly 
dependent on the modeling technique used. Many 
choices are available, but the one employed was the 
Method of Moments (MM), (see Roger F. Harrington, 
“Field Computation by Moment Methods”, The Mac- 
millan Company, New York, 1968.) This modeling pro- 
cedure has been shown to be accurate under properly 
controlled conditions. The state of the art in electro- 
magnetic modeling is still developing, and best results 
are obtained for simple, perfectly conducting objects. 
Such simple objects would include plates, ellipsoids, 
polygonal solids, and perhaps a rudimentary satellite 
model. However, more powerful computers and micro- 
wave equipment should be available in the future, mak- 
ing much more accurate modeling possible for more 
complex targets. Radar systems now exist for measuring 
range, range rate, scattered amplitude, and scattered 
phase in various polarizations. Research is progressing 
on systems that will construct images of remote objects 
using wideband microwave measurements taken over a 
variety of scattering angles. These wideband radar im- 
aging systems would be an ideal complement to optical 
imaging in space. They are, however, extremely com- 
plex and require much signal processing. In addition, 
the resulting image is often difficult to interpret, yield- 
ing intense blobs at the target’s so-called scattering 
centers. A wideband radar image would certainly be 
useful to a space robot, but is difficult and expensive to 
obtain. Instead, the present invention utilizes shape 
information that is available from the returns of simple, 
currently available, microwave radar systems. 

Accordingly, it is, therefore, an object of the inven- 
tion to employ the use of simple radar to obtain polar- 
ized RCS for shape determination. 

It is a further objective of the invention to use itera- 
tive error minimization. 

It is a still further object of the invention to integrate 
an optical image and radar data. 

It is a yet further object of the invention to use an 
optical image to provide a first approximation of the 
target shape. 

It is an even further object of the invention to identify 
novel shapes without the requirement of matching an 
image from a preexisting library. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The objects, advantages and features of the invention 
will become more apparent by reference to the draw- 
ings which are appended hereto and wherein an illustra- 
tive embodiment of the invention is shown, of which: 

FIG. 1 is a schematic diagram of the system. 

FIG. 2 is a symbolic representation of the system for 
use in a working space robot. 

FIG. 3 illustrates a dimensioned, perfectly conduct- 
ing, rectangular fiat plate as used in reconstruction 
problem. 

FIG. 4 is a plan view diagram illustrating the flat 
plate edge reconstruction problem. The right-hand 
edge of the plate is lost in shadow and will be deter- 
mined using the method of the invention. 
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FIG. 5 is an illustration of the top portion of an arbi- 
trary surface whose specular region is to be recon- 
structed. The peripheral details are known from the 
optical image, and the central portion is recovered using 
the RCS data by the method of the invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

The Radar Scattering Cross-section (RCS) is the 
ratio of scattered to incident power normalized for 10 
wavelength, range and antenna gain. The co- and cross- 
polarized RCS yield some shape information about the 
scattering object. By considering the RCS in conjunc- 
tion with an imperfect optical image, an object surface 
characterization is derived which is more reliable and 
complete than those derived from either data set taken 
alone. 

An embodiment of the invention can best be de- 
scribed in reference to FIG. 1. 

A target object of unknown shape (1) lies in free 20 
space. An object of this invention is to determine the 
shape of this target in the greatest possible detail. This is 
done by combining the outputs of an image sensor (2), 
and a radar measurement system ( 10 - 14 ). The outputs 
of these sensors are combined by image processors ( 4 , 25 
6 ), and a computer ( 16 ) for eventual display on a CRT 
( 18 ), or use by a Robot ( 19 ). 

The image sensor (2) is composed of one or more 
cameras which consist of an aperture and a light sensi- 
tive element such as a charge-coupled device (CCD) 30 
array. The light sensitive element converts the sensed 
image into an analog video signal. Multiple cameras are 
required for stereo imaging, otherwise a single camera 
is used. The analog video signal is transferred to the 
low-level image processor ( 4 ) via a coaxial cable ( 3 ). 

The low-level image processor ( 4 ) is an electronic 
circuit that may be implemented as a general purpose 
computer with specialized programming or as a series 
of specialized circuit boards with a fixed function. The 
low-level image processor ( 4 ) collects the analog video 40 
signal for each frame and converts it to a digital form. 
This digital image is an array of numbers stored in the 
memory of the low-level image processor ( 4 ), each of 
which represents the light intensity at a point on sensing 
element of camera (2). When stereo imaging is used, this 
may be done for multiple video signals simultaneously. 
The low-level image processor ( 4 ) may also perform 
certain filtering operations on the digital image such as 
deblurring, histogram equalization, and edge enhance- 
ment. These operations are well-known and fully de- 
scribed in the literature, see for example, K. R. Castle- 
man, “Digital Image Processing”, Prentice-Hall, Engle- 
wood Cliffs, N.J., 1979. 

The digital image, thus enhanced, is transferred via a 
digital data bus (5) to the high-level image processor (6). 

The high-level image processor (6) may also be either 
a hard-wired circuit or a general purpose computer. 

The high level image processor (6) takes the enhanced 
digital image and attempts to extract shape information 
from it by various means including shape-from-shading, 
or in the case where multiple cameras are used, stereop- 
sis or photometric stereo. These are also well-known 
operations and are described in the literature, (for exam- 
ple, see Berthold Klaus Paul Horn, “Robot Vision”, 
Cambridge, Mass., MIT Press, 1986.) When multiple 
cameras are used in the image acquisition and process- 
ing system (2-5), the multiple images are combined at 
the high-level image processor (6). The high-level- 
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image processor (6) produces an incomplete surface 
model of the unknown object (1). The surface model is 
a list of digitally stored data, each of which consists of 
three numbers that are the x, y, and z locations of a 
5 point on the object’s surface. The incompleteness of the 
surface model may result from regions on the object’s 
surface that are, for some reason, not understood by the 
high-level image processor (6). The incomplete surface 
model is passed on to the initializer (8) by a digital data 
bus (7). 

The initializer (8) is a general-purpose computer or 
digital circuit that “fills in” the portions of the surface 
model left incomplete by the high-level image processor 
(6). The unknown areas of the surface model are com- 
15 puted by surface functions such as B-splines that depend 
on some numerical parameter p. This surface approxi- 
mation technique is well-known and is described in 
books such as “Geometric Modeling” by Michael E. 
Mortenson, Wiley, New York, 1985. 

The surface functions are represented digitally in a 
form that both the initializer (8) and the computer (16) 
understand. The surface functions along with the in- 
complete surface model are passed on to the computer 
(16) by a digital data bus (9). The computer (16) will 
determine the correct value of the parameter p in the 
manner hereafter described. 

Concurrently with collection of the images by the 
cameras (2), a radar cross-section (RCS) of the un- 
known object (1) is being measured by the radar system 
( 10 - 14 ). The radar system consists of a radar processor 
( 14 ), antennas ( 10 , 11 ) and waveguides ( 12 , 13 ). The 
radar processor ( 14 ) is a widely-available device, and all 
of the functions described here can be performed by a 
unit such as the Hewlett-Packard 8510 Network Analy- 
35 zer. The method by which they may be performed is 
described-in Hewlett-Packard’s product note #8510-2. 
The radar processor ( 14 ) generates a microwave signal 
that is transmitted along the waveguide (12) and radi- 
ated by the transmitting antenna ( 10 ). The electromag- 
netic field is diffracted by the object (1) and collected 
by the receiving antenna ( 11 ). The diffracted signal is 
transmitted back to the radar processor ( 14 ) by a wave- 
guide ( 13 ). The radar processor ( 14 ) computes the RCS 
of the unknown object (1). The RCS is represented 
45 digitally by the radar processor ( 14 ), and transferred to 
the computer ( 16 ) by a digital data bus ( 15 ). 

The computer ( 16 ) performs the comparisons and 
iterations using two pieces of widely available software. 
The first is the MINPACK package of non-linear mini- 
50 mization programs published by Argonne National 
Laboratory, and the second is the Numerical Electro- 
magnetic Code (NEC) available from Ohio State Uni- 
versity. The NEC code generates theoretical approxi- 
mations of the RCS of the object (1) using the surface 
55 model produced by the initializer (8). The MINPACK 
program “lmdifO” uses these approximate RCSs to 
compute the correct value of the parameter p by an 
iterative scheme known as “nonlinear least squares”. 
This method allows computation of the correct value of 
60 p by minimizing the difference between the observed 
RCS acquired by the radar system ( 10 - 14 ) and the theo- 
retical RCS computed by the NEC code from the in- 
complete surface model. Using the correct value of p, 
along with the incomplete surface model and surface 
65 functions from the initializer (8), the computer gener- 
ates a complete surface model. 

The Appendices include exemplary listings showing 
the manner in which the invention was implemented in 
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specific embodiments. As with all computer programs, 
they must be written for the particular hardware and 
carefully debugged. The appendices are useful exam- 
ples of one implementation. 

Appendix I contains listings of exemplary program 5 
for determining surface parameter vector from an ob- 
servation vector and for computing the polarized RCS 
for a closed ellipsoid having specular patch and for 
minimizing the sum of squares of the vector. 

Appendix II contains listings of exemplary program 10 
to compute the polarized RCS for an ellipsoid with 
specular patch computed with a B-spline fit. 
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Appendix III contains listings of exemplary program 
to compute a specular patch for a closed ellipsoid. 

Appendix IV contains listings of exemplary program 
to generate a grid on an ellipsoid surface 

Appendix V contains listings of exemplary programs 
to do a point matching solution to the MFIE on a grid 
on n points p, with normals in the array n. 

While a particular embodiment of the invention has 
been shown and described, various modifications are 
within the spirit and scope of the invention. The ap- 
pended claims are, therefore, intended to cover all such 
modifications. 


Appendix I 


#include <math.h> 

#def ine PI 3.141592654 
tdefine K (2 . 0*PI/1 .28) 

#define B 3.25 
#define N 2 
tdefine M 3 

tdefine LWA (M*N+5*N+M) 
tdefine MAXFEV 100 

tdefine MODE 1 /* variables scaled internally*/ 

tdefine FACTOR 100.0 

tdefine NPRINT -1 

tdefine TOL 3.7253e-09 

tdefine THRESH le-8 

tdefine PERT (1.25/16.0) /* perturb by lambda over 16 */ 

tdefine MAXPERT 1 

tdefine TRUE 1 

tdefine FALSE 0 

tdefine BIGNUM 2147483647 

tdefine SCALE1 1.0 

tdefine SCALE 2 100.0 

tdefine SCALE 3 10.0 


double sigmao [M] , enorm__ ( ) , gauss ( ) ; 
double k,phix; 
int fcncnt; 


double dpmpar_ (i) 
int *i; 
i 

double result; 
if (*i — 1) { 
result * 1.0; 

while ( (result /2 . 0 +1.0) !* 1.0) 

result /* 2.0; 
return (result) ; 

} 

else if (*i " 1) 
return (TOL) ; 
else 

return (HUGE) ; 


function (m, n,x, fvec, if lag) 
int *m, *n, *if lag; 
double x [ ] / fvec [ ] ; 

{ 

double dummy; 
fcncnt++; 

rcs_ellip(1.0,0.0, &fvec[0] , &fvec[l] ,x[0],x[l]) ; 
fvec [0] *= SCALE1; fvec[0] — SCALE1* sigmao [ 0] ; 
fvec [1] *- SCALE 2 ; fvec[l] — SCALE2*sigmao [ 1] ; 
rcs_ellip (0 . 0/ 1 . 0, fidummy, &fvec [2] , x [0] , x[l] ) ; 
fvec [2] *~ SCALE3; fvec [2] SCALE3*sigmao [2] ; 
if (fcncnt > MAXFEV) 

*iflag « -1; 

} 
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/* 

* A program to determine some surface parameter vector of length N given 

* an observation vector of length M. The routine function () calls a program 

* rcs_ellip() which computes the polarized RCS for a closed ellisoid with a 

* B-spline-f it specular patch. The parameters are z-heights for the internal 

* knots of the B-spline patch. Subroutine lmdifl () is the Levenberg-Marquardt 

* algorithm for minimizing the sum of squares of the vector computed by 

* function ( ) - 
*/ 




main (argc, argv) 
int argc; 
char * argv [ J ; 

{ 

static int n-N,m-M, Iwa - LWA, narg - 1; 

int i, j, info, nfev, iwa (N) , ( *fcn> () ; 

int terminate, npert; 

double tol, resid, noise, z; 

double x (N) , fvec (M) , wa (LWA] ; 

double enorm; 

/* initialize x * / 
for (i«0; i<N; i++) ( 

x[i] - atof (argv (narg++) ) ; 
l 

fen - function; 
for (i-0;i<M;i++) { 

sigmao[i] - atof (argv (narg++) ) ; 

1 

/ * add noise to observations */ 
noise - atof (argv (narg++] ) ; 
for (i-0 ; i<M; i++) 

sigmao(i] +• noise*gauss ( ) ; 
tol - TOL; 

/* call lmdiff */ 
fenent - 0; 

lmdif 1_ (fen, fim, tn, x, fvec, Ctol, 4 inf o, iwa, wa, filwa) ; 
resid - enorm_(£m, fvec) ; 

printf ("\x [0 ] - %g, xfl] - %g, residual » %g\n", x [0] , x [ 1) , resid) ; 
printf("%d function evaluations" , fenent) ; 


Appendix II 


•include "mommeth.h" 

•define LAMBDA 1.25 
•define D (A) (delphi+A) 

•define MG 5 /* number of radial grid lines */ 

•define NG 4 l* radial distance division (NG-1 points on each line) */ 
•define NPTS (MG* (NG-1) +1) 


/* 

% A program to compute the polarized RCS for an ellipsoid with specular patch 
* computed with a B-spline fit. 

V 

rcs_ellip (hxi, hyi, sigma xx, sigma xy, patrol, pa rm2) 

double hxi, hyi; /* the incident magnetic field magnitudes */ 

double *sigmaxx, *sigmaxy; 

double parml, parm2; 

( 

static THREEVEC p{NPTS), norro(NPTS); 

static double alpha (NPTS), xO - 0.0, yO ~ 0.0, zO ■■ 25.0, a - 0 . 75 ‘LAMBDA, b * 0 . 5 * LAMBDA, 
static int n - NPTS; 

CMPLXTHREEVEC *delphi; 

complex kernel [3*NPTS] [3*NPTS] , h[3*NPTS], 1[3*NPTS), htrans (3*NPTS) , astarx [3*NPTS) , asta 
double d, partzx, partzy, xbound, ybound; 
int i,j,k; 

delphi - (CMPLXTHREEVEC *)malloc(n*n*sizeof(CMPLXTHR£EVEC)); 
n*-closed_e_grid (a,b,c,MG,NG, p) ,* 

/* compute'”norroals */ 
for ( j“0; j<n-l; j++) { 
alpha [ j] - 1.0; 

partzx *• c*c*p ( j) .*/ <a*a*faba (p[ j] .z) ) ; 
partzy - c*c*p[ j) .y/ (b*b*fabs (p( j] .z) ) ; 

norm(j).x - partzx/sqrt (1 . 0 + partzx*partzx + partzy*partzy) ; 
normt j] .y - partzy/sqrt (1 . 0 + partzx*partzx + partzy*partzy) ; 
if (pC j] -2 > 0) 

norm(j).z - 1.0/sqrt(1.0 + partzx*partzx + partzy*partzy) ; 
else 

norm(j].z - -1.0/sqrt (1 .0 + partzx*partzx + partzy*partzy) ; 

) 

xbound - a * 2. 0/3.0; ybound - b * sqrtd.O - (4. 0/9.0)); 
compos pec_patch (a, b, c, xbound, ybound, parml, parm2 , p, norm) ; 
d - z0*z0; 

gradient^ phi (p, delphi, n, 2 . 0*PI/LAMBDA) ; 
build_fc_matrix (norm, delphi, kernel, n) ; 
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build_V^array (h,p,nonn,n, 2 .0*PI/LAMBDA, x0, yO, zO,hxi) hyi) ; 
build_a_array (aatarx, aatary,p, norm, 2 . 0*PI/LAMBDA,d, n) ; 
aolve_for_l (kernel, aatary, l,n) ; 
hermtranaThfhtrana, 3*NPTS, 1) ; 
cmmult (htrana, 1, «hac, 1, 3*NPTS, 1) ; 

*aigmaxx ■» (hac.re^hac.re+hac.ira^hac.ira); 
solve_for_l (kernel, aatarx,l,n) ; 
herrotrana (h, htrana, 3*NPTS, 1) ; 
cmmult (htrana, 1, fihac, 1, 3*NPTS, 1) ; 

•aigmaxy (hac .re*hac.re+hac .im*hac.im) ; 
free (delphi) ; 

} 


Appendix III 


•define ZH (A, BJ c*aqrt<1.0 - pkljA] IB] .x*pkl JAHB] .x/ <a*a) - pkl(A](B) .y*pkl(A) (B] .y/ (b«b) 


/* 

• Thia program aomputea a apecular patch for a cloaed ellipaoid. The surface 

• patch ia controlled by the array PklUU- 

*/ 

comp apec oatch (a,b, c, xbound, ybound,parml,parm2,p, norm) 
double a,E,c, xbound, ybound, pa rml, pa rra2; 

THREEVEC p[NPTS J ,norm(NPTS] ; 

( 

THREEVEC pklf3)f4]; 
double u,w, theta; 
int i; 

pkl{0](0].x - -xbound; pkl[0][0].y - ybound; pkl[0](0].z • ZH<0,0); 

pkllOJllJ.x - -xbound/3 . 0; pkl(0][l].y - ybound; pkl[0)[l].z - ZH<0,1); 

pklC0](2].x - xbound/3. 0; pklf0](2).y - ybound; pklI0]r2).z - ZH(0,2); 

pkl [0] [3] .x - xbound; pkl(0]I3].y - ybound; pkl[0][3].z - ZH(0,3); 

pklUHOJ.x - -xbound; pkl[l)[0].y - 0.0; pkl[l)I0].z - ZH(1,0); 

pklUHD.x - -Xbound/3. 0; pklUHU.y - 0.0; pklllHl].* - parml; 

pkl [1] 12] . x - xbound/3. 0; pkl[l][2].y - 0.0; pkl[l]{2].z - parm2; 

pkl [1] (3] .x - xbound; pklfl](3J.y - 0.0; pfclClJPJ.z - ZH(1 ,3); 

pkl [2] [0] .x - -xbound; pklt2](0].y - -ybound; pkl(2](0].z * ZH(2,0>; 

pkl (2] [1] .x - -xbound/3. 0; pkl[2)[l].y - -ybound; pkl[2](l].z - ZH<2,1); 

pkl [2] 12] .x - xbound/3. 0; pkl[2]{2).y - -ybound; pklI2][2].z - ZH(2,2); 

pkl [2] [3] . x - xbound; pklt2](3).y - -ybound; pkl(2]t3].z - ZH<2,3); 

for (i-0;i<MG; i++) { 
theta - i*2 . 0 *PI/MG; 
vr - 0.5 + 0 .5*coa (theta) ; 
u - 0.5 - 0 . 5*ain (theta) ; 
findpt (u, v, tpi i* (NG-1) ] ,pkl) ; 
find normal (u, w, tnorm(i* (NG-1) ) ,pkl) ; 

} 

findpt (0 .5, 0 . 5, fip (NPTS-1] , pkl) ; 

find normal (0 .5, 0 . 5, tnorm (NPTS-1] ,pkl) ; 

1 


Appendix IV 

/• 

• A program to generate a grid on an ellipaoidal aurface with x, y, and z 

• axea equal to a,b, and c respect ively . The grid ia radial, with k radiating 

• linea of 1-1 points on each. A grid point ia alao placed at the center. 

• The total number of pointa generated ia (k # (1-1) ) 4-1. 

*/ 

•include "mommeth.h" 

•define EDGEDIST 0.9 /* diatance from center fo edge pointa */ 

•define BACKDIST 0.5 /* diatance from center for ring of pointa on backaide */ 

•define P (A) (p+(A) ) 

cloaed_e_grid (a,b,c,k, l,p) 
double a,b,c; 
int k, 1; 

THREEVEC *p; 

I 

int i, j,pctr; 
double theta; 
pctr - 0; 
for (i-0;i<k;i++) f 

theta - i*2*PI/k; 
for ( j-1; j<l; j++) ( 
if (j — 1-1) { 

P (pctr) ->x - BACKDIST*a*coa (theta) ; 

P (pctr) ->y - BACKDIST*b*ain (theta) ; 

P (pctr) ->z - - c*aqrt (1.0 - P (pctr) ->x*P (pctr) ->x/ (a*a) - P (pctr) ->y*P (pctr ) ->y 
pctr++; 

) 

elae { 
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if(j — 1-2) { /* place outer points close to edge V 
P (pet r) ->x - EDGEDIST*a*cos (theta) ; 

P(pctr)->y - EDGEDIST*b*sin (theta) ; 

1 

else { 

P (petr) ~>x - (double) j* (1 .0/ (double) 1) *a*cos (theta) ; 

P(pctr)->y - (double) j* (1.0/ (double) 1) *b*sin (theta) ; 

P(pctr)->z - c*sqrt (1 . 0 - P (petr) ->x*P (petr) ->x/ (a*a) - P (petr) ->y*P (petr) ->y/ ( 
pctr++; 

) 

) 

J 

P (petr) ->x - 0.0; 

P (petr) ->y - 0.0; 

P(pctr++)->z • c; 
return (petr) ; 

) 


Appendix V 


♦include "mommeth.h" 

♦define H (A) (h+(A)) 

♦define P (A) (p+(A)) 

♦define N (A) (norm+{A)) 

♦define DEL (A, B) (del+ (A) *n+ (B) ) 

♦define ALPHA (A) <alpha+(A)) 

♦define AX (A) (ax + (A)) 

♦define AY (A) (ay + (A)) 

♦define A1(B,C) (al+ (B) *3*n+ (C) ) 

♦define L (A) (1+(A)) 

♦define A (B,C) (a+ (B) *3*n+ (C) ) 

/* 

* A collection of programs to do a point -matching solution to the MFIE 

* on a grid of n points p, with normals in the array n. 

*/ 


build_h_array (h,p,norm,n,k,xO,yO,zO,hOx,hOy) 
complex *h; /* h array - to be computed */ 

THREEVEC *p, *norm; /* p is grid point array, norm is surface normal array*/ 
int n; /* surface grid dimensions */ 

double k; /* wavenumber */ 

double x0,y0, zO; /* source location */ 

double h0x,h0y; /* linearly polarized components of H inc */ 

{ 

int i , j ; 

complex tempi, temp2; 
for (i=0 ? i<n; i++) { 

tempi .re = 0.0; 

tempi. im = k*(P(i)->z - zO); 

cexp (fitempl, &temp2) ; /* temp2 *= exp(jkz) */ 
if (P (i) ->z > 0) { 

H(i)->re = -2.0 * N(i)->z * hOy * temp2.re; 

H (i) ->im = -2.0 * N(i)->z * hOy * temp2.im; 

H(i+n)->re =2.0 * N(i)->z * hOx * temp2.re; 

H(i+n)->im =2.0 * N(i)->z * hOx * temp2.im; 

H(i+2*n)->re = 2.0 * (N(i)->x*h0y - N(i)->y*h0x) * temp2.re; 

H (i+2*n) ->im = 2.0 * (N(i)->x*h0y - N(i)->y*h0x) * temp2.im; 

} 

else { 

H(i)->re = 0.0; 

H (i) ->im = 0.0; 

H(i+n)->re = 0.0; 

H(i+n)->im = 0.0; 

H (i+2*n) ->re =0.0; 

H(i+2*n)->im = 0.0; 


\ 


) 
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build_k_matrix (norm, del, a, n) 

THREEVEC ‘norm; /* m*n long array of normal vectors */ 

CMPLXTHREEVEC *dei.* /* the gradient of the green's function */ 
complex *a; /* the a matrix - to be determined */ 

int n; /* dimension of the surface grid */ 

t 

complex »axx, # axy, *axz, *ayx, *ayy, *ayz, *azx, *azy, *azz; 
int i, j; 

allocate^ blocks (4axx, 4axy, 4axz, 4ayx, 4ayy,4ayz, 4azx, 4azy, fiazz, n) ; 
build_blocks (axx, axy, axz, ayx, ayy, ayz, azx, azy, azz, norm, del, n) ; 
assemble^ blocks (a, axx, axy, axz, ayx, ayy, ayz, azx, azy, azz, n) ; 
f ree_blocks (4 axx, 6 axy, 4 axz, 4ayx, 4ayy, fiayz, 4azx, 4azy,4azz) ; 
for (T-0; i<3*n;i++) 

for ( j«0; j<3*n; j++) { 

A (i, j) ->re /- 2.0*PI; 

A(i, j) ->im /- 2 . 0*PI; 

J 

) 

allocate^ blocks (axx, axy, axz, ayx, ayy, ayz, azx, azy, azz, n) 
complex **axx, **axy, **axz, **ayx, **ayy, **ayz, **azx, **azy, **azz; 
int n; 

I 

•axx « (complex ♦) malloc (n*n*sizeof (complex) ) ; 

♦axy - (complex *) malloc (n*n*sizeof (complex) ) ; 

•axz - (complex *)malloc (n*n*sizeof (complex) ) ; 

♦ayx • (complex *) malloc (n*n*sizeof (complex) ) ; 

♦ayy » (complex *) malloc (n*n*sizeof (complex) ) ; 

♦ayz - (complex * j malloc (n*n*sizeof (complex ) ) ; 

♦azx » (complex *) malloc (n*n*sizeof (complex) ) ; 

♦azy - (complex *) malloc (n*n*sizeof (complex) ) ; 

♦azz - (complex ♦) malloc (n*n*sizeof (complex) ) ; 

I 

buildjblocks (axx, axy, axz , ayx, ayy, ayz, azx, azy, azz, norm, del, n) 
complex *axx, *axy, *axz, *ayx, *ayy, *ayz, *azx, *azy, *azz; 

THREEVEC *norm; 

CMPLXTHREEVEC *del; 
int n; 

( 

int i,j; 

for (i-0;i<n;i++) 

for (j-0; j<n; j++) ( 

if (N ( j) ->z !- 0.0) { 

(axx+i*n+ j) ->re - (N(i) ->y* (DEL (i, j) ->re) ,y+N(i)->z*DEL(i, j)->re.z) 
(axx+i*n+ j) ->im - <N (i) ->y*DEL (i, j) ->im.y+N(i) ->z*DEL (i, j) ->im. z) /N 
(axy+i*n+ j ) ->re - -N(i) ->x»DEL(i, j) ->re.x/N( j) ->z; 

(axy+i*n+ j ) ->im - -N(i) ->x*DEL(i, j)->im.x/N( j)->z; 

(axz+i*n+j) ->re - -N(i) ->z*DEL(i, j) ->re.y/N ( j) ->z; 

(axz+i*n+j) ->im - -N(i) ->z*DEL(i, j)->im.y/N( j)->z; 

(ayx+i*n+ j) ->re - -N(i)->x*DEL(i,j)->re.y/N(j)->z; 

( ayx+i *n+ j ) -> i® - -N(i) ->x*DEL(i, j) ->im.y/N( j) ->z; 

(ayy+i*n+j)->re - (N(i)->z*DEL(i,j)->re.z+N(i)->x*DEL(i,j)->re.x)/.\* 
(ayy+i*n+j)->im - <N(i) ->z*DEL(i, j) ->im.z+N(i)->x*DEL(i, j)->im.x) /:: 
(ayz+i*n+ j) ->re - -N(i) ->z*DEL(i, j) ->re.y/N< j) ->z; 

(ayz+i*n+ j) ->im - -N (i) ->z*D£L(i, j) ->im.y/N ( j) ->x; 

(azx+i*n+j) ->re - -N (i) ->x*DEL (i, j) ->re . z/N ( j) ->z; 

(azx+i*n+j) ->im - -N(i) ->x*DEL(i, j) ->im.z/N( j) ->z; 

(azy+i*n+ j) ->re - -N(i)->y*DEL(i, j) ->re.z/N( j)->z; 

(azy+i*n+j) ->ira - -H(i) ->y*DEL(i, j) ->im.z/N(j)->z; 

(azz+i*n+ j) ->re - (N (i) ->x*DEL(i, j) ->re .x+N (i) ->y*DEL(i, j) ->re.y) /N 
(azz+i*n+ j) ->im - (N (i) ->x*DEL(i, j) ->im.x+N (i) ->y*DEL(i, j) ->im.y) /N 


assemble_blocks (a, axx, axy, axz, ayx, ayy, ayz, azx, azy, azz, n) 
complex »a, *axx, "axy, *axz, *ayx, *ayy, *ayz, *azx, *azy, *azz; 
int n; 

{ 

int i, j; 

for (i»0;i<nri++) 

for ( j«0; j<n; j++) { 

A(i, j) ->re - (axx+i*n+ j) ->re; 

A (i, j) ->im - (axx+i*n+j) ->im; 

A (i, j+n) ->re • (axy+i*n+ j) ->re; 

A (i, j+n) ->iro - (axy+i*n+ j) ->im; 

A(i, j+2*n)->re - (axz+i*n+ j) ->re; 

A(i, j+2*n)->im - (axz+i*n+ j) ->im; 

A(i+n, j) ->re - <ayx+i*n+ j) ->re; 

A(i+n, j)->ira - (ayx+i*n+j) ->im; 

A (i+n, j+n) ->re - (ayy+i*n+j)~>re; 

A(i+n, j+n)->im - (ayy+i*n+ j) *>im; 

A(i+n, j+2*n)->re - (ayz+i*n+j) ->re; 
A(i+n, j+2*n)->im - (ayz+i*n+j)->im; 
A(i+2*n, j)->re » (azx+i*n+ j) ->re; 
A(i+2*n, j) ->im - (azx+i*n+ j) ->im; 
A(i+2*n, j+n)->re - (azy+i*n+ j) ->re; 
A(i+2*n, j+n)->im - (azy+i*n+ j) ->im? 
A(i+2*n, j+2*n) ->re - (azz+i*n+ j) ->re; 
A(i+2*n, j+2*n) ->im - (azz+i*n+ j) ->im; 

) 
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free blocks (axx, axy, axz, ayx, ayy, ayz, azx, azy, azz) 

complex **axx, **axy, **axz, **ayx, **ayy, **ayz, **azx, **azy, **azz; 

{ 

f ree (*axx) ; 
free(*axy); 
free (*axz) ; 
free (*ayx) ; 
free (*ayyj ; 
free (*ayz) ; 
free (*azx) ; 
f ree (*azy) ; 
free (*azx) ; 

J 

/* 

• A program to compute the gradient of the free space vector Green's 

• Function, ri represents the observation point# and rj the source point# or 

• variable of integration. 

*/ 


gradient_phi (p,del, n, k) 


THP.EEVEC *p; /• n long array of surface points */ 

CM? LXTHREE VEC *del; /* n by n array of complex# three component elements */ 
int n; 

double k; / * wavenumber •/ 


{ 


int 1,3 
double 
complex 
for (i-0 


dx, dy, dz, d; 

tempi , temp2, temp3 ; 
; i<n ; i + + ) 
for ( j-0; j<n; j++) { 


if (i 


else 


j) ( 

dx - P { i ) ->x - P ( j ) ->x; 
dy - P (i) ->y - P(j)->y; 
dz - P (i) ->z - P { j) ->z; 
d - sqrt(dx*dx + dy*dy + dz*dz) 
tempi. re “ 1.0/(d*d*d); 
tempi. im - k*1.0/(d*d); 
temp2.re - 0.0; 
temp2.im « -k*d; 
cexp (*temp2, ttemp3) ; /* temp3 - 
cmult (6terop3, £templ r 4temp2) ; /* 
DEL (i, j) ->re .x - dx*temp2.re; 
DEL (i, j) ->im.x - dx*temp2.im; 
DEL (i, j) ->re .y - dy*temp2.re; 
DEL (i, j) ->im.y « dy*temp2.im; 
DEL (i, j ) ->re . z - dz*temp2.re; 
DEL(i, j) ->im.z - dz*temp2.im; 

J 


DEL (i, j ) ->re .x - 0.0 
DEL (i, j) ->im.x - 0.0 
DEL (i# j) ->re .y - 0.0 
D£L(i, j) ->im.y - 0.0 
DEL (i, j ) ->re . z - 0.0 
DEL Ci f j) ->im.z -0.0 


exp {- jkd) */ 
temp2 - tempi *temp3 


1 


*/ 


build_a_array (ax, ay,p r norm, k,d,n) 
complex *ax,*ay; 

THREEVEC *p, ‘norm; 
double k,d; 
int n; 

complex tempo , tempi , temp2 , temp3 , temp4 ; 

int i; 

/• 

• Perform numerical integration 

•/ 

temp3.re - 0.0; temp3.im * 0.0; 

temp4.re « 0.0; temp4 . im » 0.0; 

/* initialize a arrays */ 

f or ( i*0 ; i<3 *n; i++) { 

AX (i) ->re - 0.0; 

AX < i) ->im - 0.0; 

AY ( i ) -> re - 0.0; 

AY ( i ) ->im - 0.0; 

\ 

t or ( i*0; i<n; i + + ) ( 

tempo. re - cos (k*P (i) ->z) ; 
tempo . im - -sin (k *P (i) ->z) ; 
temp3.re * tempo . re/N { i) ~>z; 
temp3.im - tempo . im/N <i) ->z; 
temp4.re - tempo . re/N (i) ->z; 
temp4.im - tempO . im/N <i) ->z; 
tempO.re ■ 0.0; 
tempO. im - k/ (4.0 *PI *d) ; 
tempi. re - cos(-k*d); 
tempi. im - sin<-k*d); 
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emu It (£ tempo, i tempi , & temp2) ; 
cmult ( £ temp3 , fctemp2, AX (i) ) ; 
cmult Utemp4, £temp2, AY (i+n) ) ; 

l 


* A program to solve for the surface currents given a complex kernel matrix 

* (I-a) - al+ja2, and an h vector h«hl+jh2. We wish to solve the equation 

• (I-a)'l-h, where x - xl+jx2. 

• All dimensions are 3*n. 

•/ 


solve_for_l (a, h, i,n) 
complex *a,*h,*l; 
int n; 
t 

complex 
int *ipvt; 

double rcond; /* work space for Unpack */ 
int i t j; 

int Ida, nn, job; /* constants for linpack */ 

/* 

• Allocate space for the temporary arrays 
V 

al - (complex *) malloc (3*n*3*n*sizeof (complex ) ) ; 
z - (complex *)malloc (3*n*sizeof (complex) ) ; 
ipvt- (int *) malloc (3*n*sizeof (int) ) ; 

/* compute and transpose al matrix al - (I-A) transpose */ 
for (i-0; i<3*n; i++) 

for ( j-0; j<3*n; j++) { 
if (i!«j) 

Al (i, j) ->re - -A(j,i)->re; 

else 

Al (i, j) ->re - 1.0 - A(j,i)->re; 

Al (i, j) ->im - A ( j, i) ->im; 

) 

/• Fill x array */ 
for (i-0 ; i<3*n; i++) { 

L ( i) ->re - H(i)->re; 

L(i)->im - H(i)->im; 

} 

Ida = 3*n; 
nn = 3*n; 

/* perform LU decomposition and condition estimation */ 
zgeco_(al, &lda y &nn f ipvt, &rcond, z) ; 
if (rcond + 1.0 == 1.0) { 

printf(” a is illconditioned, rcond = %g\n%", rcond) ; 
return; 

} 

job * 0; 

/* solve complex system of equations */ 
zgesl_(al, &lda, &nn, ipvt, 1, & job) ; 

/* free space */ 
free (al) ; 
free (z) ; 
free (ipvt) ; 
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We claim: 

1. A method for determining the shape of a remote 
surface by sensor fusion using incomplete data from a 
plurality of sensors, comprising the steps of 

(a) obtaining from a first sensor a first electronic 60 
signal representative of the surface, 

(b) constructing a first electronic model of the surface 
using the first electronic signal, 

(c) using first electronic model as a guide, construct- 
ing for a second sensor a predicted electronic signal 
also representative of the surface, 

(d) obtaining from the second sensor an actual elec- 
tronic signal representative of the surface, 


(e) minimizing the difference between the predicted 
electronic signal and the actual electronic signal, 

(0 constructing an additional electronic model of the 
surface using the result of step (e) as a guide, 

(g) using the additional electronic model of the sur- 
face in constructing an additional predicted elec- 
tronic signal for the second sensor, 

(h) obtaining from the second sensor an additional 
actual electronic signal representative of the sur- 
face, 

(i) minimizing the difference between the additional 
predicted electronic signal and the additional ac- 
tual electronic signal, 



5,005,147 


31 

(j) repeating steps (f) through (i) until the difference 
between the final predicted electronic signal and 
the actual electronic signal is below a predeter- 
mined threshold level whereby, the shape of the 
remote surface, represented by the final predicted 5 
electronic signal, is determined more accurately 
than can be done with either the first or second 
sensor alone, and 

(k) displaying the shape of the remote surface. 

2. The method of claim 1, wherein the first sensor is 
an optical device and the second sensor is a radar de- 
vice. 

3. The method of claim 2, wherein the optical device 
is a camera and the radar device is a microwave radar. 

4. The method of claim 1 wherein a model is con- 
structed using image shaping techniques, the theoretical 
cross-sections are generated using moment-methods 
techniques, and the difference between the theoretical 
and actual cross-sections are minimized by use of the 
“lmdifO” program from the MINPACK group of pro- 
grams published by Argonne National Laboratories. 

5. The method of claim 4 wherein the moment- 
method technique is the Numerical Electromagnetic 
Code published by Ohio State University. 

6. The method of claim 5 wherein the image shaping 
technique is a shape-from-shading technique. 

7. The method of claim 5 wherein the image shaping 
technique is a photometric stereo technique. 

8. The method of claim 5 wherein the image shaping 30 
technique is a stereopsis technique. 

9. The method of characterizing the incompletely 
observable surface shape of a physical object compris- 
ing the steps of 

(a) obtaining an initial optical image of the object, 35 

(b) extracting from the initial optical image of the 
object occluding contours by thresholding the. 
initial optical image, 

(c) deriving from the initial optical image a partial 
shape description for the object by shape-from- 40 
shading techniques, 

(d) computing a set of predicted values for RCS from 

digital representation of initial optical image using 
method of moments, 45 

(e) obtaining a set of polarized radar scattering cross- 
sections from the object, 

(f) minimizing the difference between the set of polar- 
ized radar scattering cross-sections and the set of 
predicted values for RCS, by linear least squares 5Q 
technique, to achieve a refined surface description, 

(g) repeating steps (d) through (f), using each succes- 
sively refined surface description in lieu of the 
initial surface description, until the difference ob- 
tained in step (0 is below a predetermined thresh- 55 
old level whereby, the shape of the physical object 
surface, represented by the refined surface descrip- 
tion, is determined more accurately than either the 
optical image or radar scattering cross sections 
alone. 

10. A method of constructing a representation of the 
surface of a physical object by sensor fusion of an in- 
complete optical image of the surface and radar scatter- 
ing cross-sections of the object comprising the steps of 

(a) obtaining an optical image of the object 

(b) obtaining an approximate shape of the object from 
the optical image, 

(c) enhancing the edges of the optical image of the 
object, 
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(d) thresholding the edge-enhanced optical image, 

(e) extracting the visible portions of the edges of the 
object from the thresholded optical image, 

(f) generating a surface shape model of the object 
from the visible portions of the object, 

(g) generating theoretical radar cross-sections* of the 
object from the model of the object, 

(h) obtaining actual radar cross-sections of the object, 

(i) minimizing the difference between the theoretical 
cross-sections and the observed cross-sections, by 
refining the surface shape model of the object 
whereby, the shape of the physical object surface, 
represented by the surface shape model, is deter- 
mined more accurately than either the optical 
image or radar scattering cross sections alone. 

11. The method of claim 10 wherein the approximate 
shape of the object is obtained by stereopsis techniques, 
the model of the object is obtained by use of a Geomet- 
rical Theory of Diffraction scattering technique, and 

lK) minimizing the difference between theoretical and ac- 
tual cross-sections is by use of the “lmdifO” program 
from the MINPACK group of programs published by 
Argonne National Laboratories. 

12. Apparatus for determining the shape of a space 
5 object by fusing the optical and microwave signals from 

the object comprising 

(a) optical sensing means for producing an electrical 
signal representative of the object, 

(b) low-level image processing means for enhancing 
the electrical signal, 

(c) high-level image processing means for extracting 
shape information from the electrical signal and for 
producing a first, incomplete surface model repre- 
sentative of the shape of the object, 

(d) initializing means for receiving the surface model 
and for filling in incomplete portions of the surface 
model, 

(e) microwave receiving means for receiving micro- 
wave signals from the object, 

(f) microwave processing means for producing ob- 
served radar scattering cross sections of the object 
from the microwave signals, 

(g) computing means for producing theoretical ap- 
proximations of the radar scattering cross sections 
of the object using the surface model produced by 
the initializing means, and for generating a refined 
surface model by comparing the observed radar 
scattering cross sections with the theoretical radar 
scattering cross sections and modifying the surface 
model to minimize the difference between the theo- 
retical radar scattering cross sections and the ob- 
served radar scattering cross sections whereby, the 
refined surface model thus produced is a more 
accurate representation of the shape of the space 
object than either the optical signal or the micro- 
wave signal, and 

(h) display means for displaying the shape of the 
object. 

60 13. The apparatus of claim 12 wherein a final, com- 

plete surface model is produced by successively and 
iteratively refining the surface model, producing theo- 
retical radar scattering cross sections based on the re- 
fined surface model, comparing the theoretical radar 
65 scattering cross sections with the observed radar scat- 
tering cross sections, and modifying the surface model 
to minimize the difference between the theoretical and 
observed radar scattering cross sections. 

14. The apparatus of claim 12 wherein the optical 
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sensing means comprises a plurality of optical devices 
and the low-level processing means comprises stereo 
imaging techniques and wherein the low-level process- 
ing means further comprises analog to digital conver- 
sion of the signal and filtering means for deblurring, 5 
histogram equalization, and edge enhancement. 

15. The apparatus of claim 12 wherein the initializa- 
tion means for filling in the incomplete surface model 
employs computation by B-splines surface functions. 

16. The apparatus of claim 12 wherein the theoretical 10 
radar scattering cross sections are produced by the 
Numerical Electromagnetic Code are available from 
Ohio State University and wherein the refined surface 
model is produced by use of the MINPACK program 
lmdifO published by Argonne National Laboratory. 15 

17. Apparatus for characterizing the shape of a space 
object by fusing optical and microwave data from the 
object, comprising 

optical image sensor means for producing video sig- 
nals representative of the object, 20 

low level image processor means for performing en- 
hancement of the video signal, said enhancement 
comprising collecting the video signals, converting 
them to digital form, and for performing image 
filtering operations on the digital signals, 25 

high level image processor means for extracting 
shape information from the enhanced video signal 
to produce an incomplete first surface shape model 
of the object, 

radar means for measuring actual radar scattering 20 
cross sections of the object, 
computing means for generating theoretically pre- 
dicted radar scattering cross sections of the object 
from the first surface shape model, for minimizing 
the difference between the theoretically predicted 25 
radar scattering cross sections and the actual radar 
scattering cross sections of the object, for refining 
the surface model as a result of the minimum differ- 


ence, and for generating a final surface shape 
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model by successive iterations of minimizing the 
difference and refining the surface model whereby, 
the final surface model more accurately character- 
izes and represents the shape of the space object 
than either the optical video signals or the radar 45 
scattering cross sections, and 
output means for outputting the shape characteriza- 
tion. 


18. Apparatus of claim 18 wherein the theoretically 
predicted radar scattering cross sections of the object 
are generated by use of the Numerical Electromagnetic 
Code available from Ohio State University. 

19. Apparatus of claim 17 wherein minimizing the 
difference between the theoretically predicted radar 
scattering cross sections and the actual radar scattering 
cross sections of the object is accomplished by use of 
the MINPACK program lmdifO published by the Ar- 
gonne National Laboratory. 

20. A method for converting electronic signals repre- 
sentative of the incomplete shape of a remote object to 
electronic signals representative of a more complete 
shape of 

(a) obtaining from a first sensor a first electronic 
signal representative of the shape of the object, 

(b) constructing a first electronic model of the shape 
using the first electronic signal, 

(c) using first electronic model as a guide, construct- 
ing for a second sensor a predicted electronic signal 
also representative of the shape, 

(d) obtaining from the second sensor an actual elec- 
tronic signal representative of the shape, 

(e) minimizing the difference between the predicted 
electronic signal and the actual electronic signal, 

(f) constructing an additional electronic model of the 
shape using the result of step (e) as a guide, 

(g) using the additional electronic model of the shape 
in constructing an additional predicted electronic 
signal for the second sensor, 

(h) obtaining from the second sensor an additional 
actual electronic signal representative of the shape, 

(i) minimizing the difference between the additional 
predicted electronic signal and the additional ac- 
tual electronic signal, 

(j) repeating steps (f) through (i) until the difference 
between the final predicted electronic signal and 
the actual electronic signal is below a predeter- 
mined threshold level whereby, the shape of the 
remote object, represented by the final predicted 
electronic signal, is determined more accurately 
than can be done with either the first or second 
sensor alone, and 

(k) displaying the shape of the remote surface. 

* * * * * 
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